Method and Vehicle Control System for Producing Images of a Surroundings Model, and Corresponding Vehicle

ABSTRACT

The invention relates to a method for producing images of a stored three-dimensional model ( 30 ) of the surroundings of a vehicle ( 20 ), said images having been corrected for perspective. A camera picture ( 32 ) is produced by a camera device ( 21 ) of the vehicle, and the produced camera picture ( 32 ) is projected onto a projection surface ( 31 ) in the surroundings model. A region ( 33 ) relevant for driving is marked in said stored three-dimensional surroundings model ( 30 ), and this marked region ( 33 ) is projected onto a corresponding projection surface area ( 35 ) of the projection surface ( 31 ). An image of the projection surface ( 31 ) with the region ( 33 ) projected onto the projection surface area ( 35 ) is produced and output by means of a virtual camera ( 72 ) that can move freely in the surroundings model ( 30 ).

The invention relates to a method for producing images of a storedthree-dimensional surroundings model, said images having been correctedfor perspective, a vehicle control system of a vehicle for producingimages of a stored three-dimensional model of the surroundings of thevehicle, said images having been corrected for perspective, and avehicle having a vehicle control system.

Surroundings models like those known, for example, from WO 2013/060323A1 can help the driver or a driver assistance system to control thevehicle. A parking assist system can, for example, display a camerapicture to the driver on a display, the expected trajectory of thevehicle being drawn in by means of auxiliary lines in said camerapicture. It is enormously important that these auxiliary lines are alsodisplayed at the correct location in the event of the camera alignmentbeing altered, in order to prevent the driver wrongly estimating thevehicle movement and colliding with objects.

It is therefore an object of the invention to provide images of asurroundings model, in which regions relevant for driving are drawn incorrectly.

This object is achieved by a method for producing images of a storedthree-dimensional model of the surroundings of a vehicle, said imageshaving been corrected for perspective, having the features of Claim 1, avehicle control system of a vehicle for producing images of a storedthree-dimensional surroundings model, said images having been correctedfor perspective, having the features of Claim 8 and a vehicle having thefeatures of Claim 15.

Preferred embodiments are the subject matter of the subordinate claims.

According to a first aspect, the invention accordingly relates to amethod for producing images of a stored three-dimensional model of thesurroundings of a vehicle, said images having been corrected forperspective. At least one camera picture is produced by a camera deviceof the vehicle and the produced camera picture is projected onto aprojection surface in the stored three-dimensional model of thesurroundings of the vehicle. A region relevant for driving is marked insaid stored three-dimensional surroundings model. The method furthercomprises the projecting of the marked region onto a correspondingprojection surface area of the projection surface, in order to identifyan image region of the camera picture projected onto the projectionsurface corresponding to the marked region as having been corrected forperspective. An image of the projection surface with the regionprojected onto the projection surface area is produced by means of avirtual camera that can move freely in the surroundings model, and theproduced image is output.

The term ‘vehicle’ denotes, in particular, a motor vehicle, inparticular a car, a motorcycle, an E-bike, a truck, a ship or boat oreven an airplane.

The term ‘image of the surroundings model’ denotes an image of aparticular spatial partial region of the surroundings model produced bymeans of the virtual camera, wherein the image depends on a position andalignment of the virtual camera.

The term ‘projection surface’ denotes a two-dimensional hypersurface inthe three-dimensional surroundings model, which can be shaped and curvedin any way desired. The projection surface can be a flat plane, aspherical shell segment, a cylindrical surface or a paraboloidal orhyperboloidal face.

The term ‘region relevant for driving’ denotes a region of thesurroundings model, which corresponds to a real spatial region of thesurroundings of the vehicle, which is important for navigating thevehicle. The real spatial region can, for example, be a spatial area, inwhich objects or obstacles which are impassable for the vehicle arelocated. The real spatial region can also correspond to a trajectory ofthe vehicle, along which the vehicle would move without the drivingparameters being altered. The real spatial region can further be aparticular range of distance from the vehicle or correspond to a parkingspace. The real spatial region can be a zero-dimensional dot-shapedregion, a one-dimensional, in particular also a curved, linear region, atwo-dimensional, possibly curved surface, or a three-dimensional volumeregion.

The fact that the marked region is projected directly onto theprojection surface area of the projection surface means that this isdirectly identified in the camera picture projected onto the projectionsurface. If the viewing direction onto the projection surface is changeddue to the movement of the virtual camera, the projection surface areanevertheless remains at the correct position such that aberrationeffects can be prevented. Such perspective errors are consequentlycorrected by the method according to the invention.

The method according to the invention improves the representation of theregion relevant for driving. Thus, it is only possible for the vehicle'sdriving function to be controlled by a driver assistance system on thebasis of the produced images once the images have been corrected forperspective since, otherwise, wrongly drawn-in regions relevant fordriving could be evaluated. The method consequently improves theprecision and accuracy of the produced images and therefore also largelyincreases safety, since the images have of course been corrected forperspective and are free of perspective errors.

According to one preferred embodiment of the method, the marked regionis projected onto the projection surface, in that a respective point ofthe marked region is imaged onto an intersection of a correspondingconnecting line with the projection surface, wherein the connecting lineconnects this point of the marked region to a predefined reference pointof the surroundings model. The reference point consequently designates apossible position of the movable virtual camera, viewed from which theregion relevant for driving is projected onto the projection surface bycentral projection. The projection surface preferably lies between thereference point and the region relevant for driving, however the regionrelevant for driving can also be located between the reference point andthe projection surface.

According to one preferred further development of the method, the cameradevice has at least one vehicle camera, wherein the reference point ofthe surroundings model corresponds to a spatial position of that vehiclecamera which produces the camera picture imaged onto the projectionsurface. Consequently, it is ensured that, due to the projecting of themarked region, said marked region is drawn in at the correct location ofthe projection surface, since an image produced from the perspective ofthe reference point reflects the real conditions with the regionrelevant for driving located at the correct position.

According to one preferred further development of the method, a cameraposition and/or camera alignment of the freely movable virtual camerais/are determined on the basis of sensor data produced by sensors of thevehicle and/or captured parameters of the vehicle. Thus, for instance inhazardous situations such as during reversing or when turning into anunclear road, the position of the virtual camera can be automaticallycontinually moved, in order to consequently output images from a viewingdirection which is convenient for the driver or which can be easilyevaluated by the driver assistance system. Possible vehicle parametersare the speed or position of the vehicle or an angular position of thewheels of the vehicle.

According to one preferred embodiment of the method, a driver assistancesystem controls a function of the vehicle on the basis of the outputimages. The vehicle functions can comprise actuating and enabling ordisabling actuators such as, for instance, indicators or side mirrors,or also semi-autonomous or autonomous acceleration, braking or steeringof the vehicle.

According to one preferred embodiment of the method, the output imagesare displayed to a driver of the vehicle on a display apparatus. Thistherefore provides the driver with an accurate overview of the vehiclesurroundings, wherein regions relevant for driving are drawn incorrectly in terms of perspective.

According to one preferred further development of the method, the regionrelevant for driving is marked in the stored three-dimensionalsurroundings model on the basis of sensor data produced by sensors ofthe vehicle and/or captured parameters of the vehicle. For example,distances of objects can be measured on the basis of radar sensors and,as a result, regions relevant for driving of the surroundings model,which correspond to spatial regions occupied by objects, are identified.

According to a further aspect, the invention comprises a vehicle controlsystem of a vehicle for producing images of a stored three-dimensionalmodel of the surroundings of the vehicle, said images having beencorrected for perspective. The vehicle control system comprises a cameradevice which is configured to produce at least one camera picture, aswell as a computing device. The computing device is configured toproject the produced camera picture onto a projection surface in thestored three-dimensional model of the surroundings of the vehicle, tomark a region relevant for driving in the stored three-dimensionalsurroundings model, and to project the marked region onto acorresponding projection surface area of the projection surface. Thecomputing device is consequently configured to identify an image regionof the camera picture projected onto the projection surfacecorresponding to the marked region as having been corrected forperspective. The computing device is further configured to produce animage of the projection surface with the region projected onto theprojection surface area by means of a virtual camera that can movefreely in the surroundings model, and to output the produced image.

According to one further development of the vehicle control system, thecomputing device is further configured to project the marked region ontothe projection surface, in that it images a respective point of themarked region onto an intersection of a corresponding connecting linewith the projection surface, wherein the connecting line connects thispoint of the marked region to a predefined reference point of thesurroundings model.

According to one preferred further development of the vehicle controlsystem, the camera device has at least one vehicle camera, wherein thereference point of the surroundings model corresponds to a spatialposition of that vehicle camera which produces the camera picture imagedonto the projection surface.

According to one further embodiment, the vehicle control systemcomprises at least one sensor of the vehicle, which is configured toproduce sensor data and/or to capture parameters of the vehicle. Thecomputing device is configured to mark the region relevant for drivingin the stored three-dimensional surroundings model on the basis of theproduced sensor data and/or on the basis of the captured parameters ofthe vehicle.

According to one further embodiment of the vehicle control system, thecomputing device is configured to determine a camera position and/orcamera alignment of the freely movable virtual camera on the basis ofthe produced sensor data and/or on the basis of the captured parametersof the vehicle.

According to one preferred embodiment, the vehicle control system has adriver assistance system which is configured to control a function ofthe vehicle on the basis of the output images.

According to one preferred embodiment of the vehicle control system, theoutput device has a display apparatus, on which the output images can bedisplayed to a driver of the vehicle.

According to a third aspect, the invention provides a vehicle having avehicle control system.

The present invention is explained in greater detail below on the basisof the embodiment examples indicated in the schematic figures of thedrawings, wherein:

FIG. 1 shows a flow chart in order to explain a method for producingimages of a stored three-dimensional surroundings model of a vehicle,said images having been corrected for perspective, according to oneembodiment of the invention;

FIG. 2 shows a top view of an exemplary scenario in order to explain themethod according to the invention;

FIG. 3 shows a schematic view of a surroundings model in order toexplain the projecting of a marked region;

FIG. 4 shows an exemplary image from a first camera perspective;

FIG. 5 shows a view of a surroundings model in order to explain theaberration effect avoided by the invention;

FIG. 6 shows an image having an aberration error;

FIG. 7 shows a schematic view of the surroundings model having a virtualcamera that can move freely;

FIG. 8 shows an exemplary image from a second camera perspective;

FIG. 9 shows a schematic block diagram in order to explain a vehiclecontrol system according to one embodiment of the invention; and

FIG. 10 shows a schematic block diagram of a vehicle according to oneembodiment of the invention.

In all of the figures, similar elements and apparatuses or respectivelyelements and apparatuses having similar functions are provided with thesame reference numerals. The method steps are numbered for reasons ofclarity. This is not, in general, intended to imply a specific temporalsequence. In particular, multiple method steps can be carried outsimultaneously. Furthermore, various embodiments can be combined witheach other at will, inasmuch as this makes sense.

FIG. 1 shows a flow chart in order to explain a method for producingimages of a stored three-dimensional surroundings model of a vehicle,said images having been corrected for perspective, according to oneembodiment of the invention.

In FIG. 2, an exemplary scenario is illustrated in a top view whichshows a vehicle 20 which is equipped with a camera device 21 which hasvehicle cameras 21 a-21 d mounted around the vehicle. Boundary posts 24as well as further road users 22, 23 are located in the surroundings ofthe vehicle 20.

In a first method step S1, at least one camera picture of thesurroundings of the vehicle 20 is produced by the camera device 21 ofthe vehicle 20.

A three-dimensional model 30 of the surroundings of the vehicle 20 isfurther provided, which is schematically illustrated in FIG. 3. Thethree-dimensional surroundings model is preferably stored in a memory ofa vehicle control system of the vehicle 20. A projection surface 31,which is for example arranged cylindrically around an origin ofcoordinates 36 of the surroundings model 30, is determined in thesurroundings model 30, wherein the origin of coordinates 36 correspondsto a position of the vehicle 20. The projection surface 31 can alsocorrespond to a spherical surface arranged around the origin ofcoordinates 36 or a partial region of a spherical surface. The shape ofthe projection surface is not limited to the indicated examples.

In a second method step S2, the produced camera picture 32 is projectedonto the projection surface 31.

In a further method step S3, a region 33 relevant for driving is markedin the stored three-dimensional surroundings model. The marking S3 ispreferably carried out automatically, in particular on the basis ofsensor data and/or parameters of the vehicle, which are preferablyproduced or respectively established by at least one sensor of thevehicle 20. Thus, a region 33 relevant for driving, which corresponds toan expected trajectory of the vehicle 20, is, for example, determined onthe basis of an angular position of the wheels of the vehicle measuredby the at least one vehicle sensor, and marked in the surroundingsmodel. The region 33 relevant for driving can be identified with colorin the surroundings model. A reference point 34 of the surroundingsmodel 30 is further identified, which corresponds to a spatial positionof that vehicle camera which produces the camera picture 32 imaged ontothe projection surface 31.

With respect to this reference point 34, the marked region 33 relevantfor driving is projected onto a corresponding projection surface area 35of the projection surface 31 in a further method step S4. The projectionsurface area 35 is an image region of the camera picture projected ontothe projection surface, which corresponds to the marked region 33, whichis consequently identified in a way which indicates it has beencorrected for perspective. To this end, an intersection 39 of theprojection surface 31 with a connecting line 38 is determined for eachpoint 37 of the marked region 33, wherein the connecting line 38connects this point 37 to the reference point 34.

In a further method step S5, an image of the projection surface 31 isproduced with the region 33 projected onto the projection surface area35 by means of a virtual camera that can move freely in the surroundingsmodel 30, and the produced image is output in a subsequent method stepS6.

FIG. 4 shows a produced first image 40 which is acquired by means of avirtual camera located at the reference point 34. Here, the markedregion 33 relevant for driving is located at the correct position anddisplays a trajectory, along which the vehicle will move if it retainsits driving parameters, to the driver of the vehicle 21.

A fundamental point of the invention is that the region relevant fordriving is projected onto the projection surface and the projectionsurface with the region projected onto the projection surface area isimaged. This differs from the image which would be produced by acquiringthe projection surface and the region relevant for driving itself.

By way of comparison, such a method is illustrated in FIG. 5 but is notpart of the invention. If the virtual camera is displaced from thereference point 34 to a predefined camera spatial point 54, said cameracaptures a picture of the projection surface 31 as well as the markedregion 33 relevant for driving, the region 33 relevant for drivingappears in a corresponding image at a displaced position 55 of theprojected camera picture 32. This is illustrated in FIG. 6 which shows asecond image 60 which is acquired with this method and which is notcomprised by the invention. It can be seen that, during thisdisplacement of the virtual camera to the camera spatial point 54, adisplacement of the imaged region 33 relevant for driving does notcorrespond to a displacement of the image section itself. Due to thisaberration effect, the trajectory of the vehicle is not displayedcorrectly in the indicated example.

In contrast thereto, FIG. 7 explains the producing of images of theprojection surface 31 with the region 33 projected onto the projectionsurface area 35 according to the invention. Here, a virtual camera 72 isdrawn in by way of example at the reference point 34 and at a furthercamera spatial point 71. Irrespective of a camera position of thevirtual camera 72 and of an alignment of the virtual camera 72, theprojection surface area 35 always appears at the same, i.e. the correct,position of the camera picture 32 on the projection surface 31.

FIG. 8 shows, by way of example, a third image 80 which has beenacquired by means of the virtual camera 72 at the further camera spatialpoint 71. As can be seen, the region 33 relevant for driving is now, inturn, located at the right position of the produced image.

As a result, the method makes it possible to correct the perspective ofthe image by preventing an aberration effect.

According to one further embodiment, the camera position and/or cameraalignment of the freely movable virtual camera 72 is/are determined onthe basis of sensor data produced by sensors of the vehicle 20 and/orparticular parameters of the vehicle 20. The camera position of thevirtual camera can thus be continually and uniformly displaced, andappropriate continual images can be produced.

According to one embodiment, the produced images are output to a driverassistance system which controls a function of the vehicle on the basisof the output images. For example, an advantageous camera view for thedriver assistance system can be selected, which is distinguished by anoptimum perspective view, as a result of which the required computingtime and computing performance of the driver assistance system forevaluation can be reduced. Starting from this camera perspective, thedriver assistance system can control the vehicle partially autonomouslyor autonomously. It is only possible for the driver assistance system tocontrol the vehicle precisely, if the regions relevant for driving aredrawn in correctly.

According to one preferred further development, the output images aredisplayed to a driver of the vehicle 20 on a display apparatus of thevehicle 20.

FIG. 9 shows a block diagram in order to explain a vehicle controlsystem 90 of a vehicle 20 for producing images of a storedthree-dimensional model of the surroundings 30 of the vehicle 20, saidimages having been corrected for perspective. The vehicle control system90 has a camera device 21 which is configured to produce at least onecamera picture 32. The camera device 21 can comprise a multitude ofvehicle cameras 21 a-21 d which allow a surround view of the environmentof the vehicle 20. The vehicle control system further has a storagedevice, on which a three-dimensional model of the surroundings of thevehicle 20 is stored. According to one further embodiment, thethree-dimensional surroundings model 30 can also be made available tothe vehicle control system by means of an interface.

The vehicle control system 90 comprises a computing device 91 which isconfigured to project the produced camera picture onto a projectionsurface 31 in the stored three-dimensional surroundings model of thevehicle. The projection surface 31 can be predefined or can bedetermined by the computing device 91 itself.

The computing device 91 is further configured to mark a region relevantfor driving in the stored three-dimensional surroundings model. To thisend, the vehicle control system 90 can optionally have at least onevehicle sensor 93 which is configured to produce sensor data and/orcapture parameters of the vehicle 20. Such vehicle sensors 93 compriseradar systems, lidar systems, optical cameras, infrared cameras or lasersystems. The region relevant for driving which corresponds, for example,to a parking space, a trajectory of the vehicle or an obstacle, can berecognized on the basis of the sensor data by the computing device 91and recorded and marked in the surroundings model 30.

The computing device 91 is further configured to project the markedregion 33 onto a corresponding projection surface area 35 of theprojection surface 31. As a result, the computing device 91 identifiesan image region of the camera picture 32 projected onto the projectionsurface 31 corresponding to the marked area 32 in a way which indicatesit has been corrected for perspective. The computing device 91 isfurther configured to produce an image of the projection surface withthe region 33 projected onto the projection surface area 35 by means ofa virtual camera 72 that can move freely in the surroundings model 30.

The vehicle control system 90 further has an output device 92 which isconfigured to output the produced image. The output device 92 can havean interface, in particular a cable connection, a USB interface or awireless interface. The produced images can, in particular, betransmitted by means of the output device 92 to further units or viacar-to-car communication to further vehicles.

Optionally, the vehicle control system 90 further comprises a driverassistance system 94 which is configured to control a function of thevehicle 20 on the basis of the output images.

According to one preferred further development, the output device 92 hasa display apparatus which is arranged in an interior of the vehicle 20and displays the output images to a driver of the vehicle 20.

FIG. 10 shows an exemplary block diagram of a vehicle 20 having avehicle control system 90 according to one embodiment of the invention.Here, the vehicle control system 90 can be one of the embodimentsindicated above.

LIST OF REFERENCE NUMERALS

20 Vehicle

21 Camera device

21 a to 21 d Vehicle cameras

22, 23 Further road users

24 Boundary posts

30 Surroundings model

31 Projection surface

32 Camera picture

33 Region relevant for driving

34 Reference point

35 Projection surface area

36 Origin of coordinates

37 Point of the marked region

38 Connecting line

39 Intersection

40 First image

54 Camera spatial point

55 Displaced position

60 Second image

71 Further camera spatial point

72 Virtual camera

80 Third image

90 Vehicle control system

91 Computing device

92 Output device

93 Vehicle sensors

94 Driver assistance system

1. A method for producing images of a stored three-dimensional model(30) of the surroundings of a vehicle (20), said images having beencorrected for perspective, having the steps of: Producing (S1) at leastone camera picture (32) by a camera device (21) of the vehicle (20);Projecting (S2) the produced camera picture (32) onto a projectionsurface (31) in the stored three-dimensional model (30) of thesurroundings of the vehicle (20); Marking (S3) a region (33) relevantfor driving in the stored three-dimensional surroundings model (30);Projecting (S4) the marked region (33) onto a corresponding projectionsurface area (35) of the projection surface (31) in order to identify animage region of the camera picture (32) projected onto the projectionsurface (31) corresponding to the marked region (33) as having beencorrected for perspective; Producing (S5) an image of the projectionsurface (31) with the region (33) projected onto the projection surfacearea (35) by means of a virtual camera (72) that can move freely in thesurroundings model (30); and Outputting (S6) the produced image.
 2. Themethod according to claim 1, wherein the marked region is projected ontothe projection surface (31), in that a respective point (37) of themarked region (33) is imaged onto an intersection (39) of acorresponding connecting line (38) with the projection surface (31),wherein the connecting line (38) connects this point (37) of the markedregion to a predefined reference point (34) of the surroundings model(30).
 3. The method according to claim 2, wherein the camera device (21)has at least one vehicle camera (21 a-21 d), and wherein the referencepoint (34) of the surroundings model (30) corresponds to a spatialposition of that vehicle camera which produces the camera picture (32)imaged onto the projection surface (31).
 4. The method according toclaim 1, wherein a camera position and/or camera alignment of the freelymovable virtual camera (72) is/are determined on the basis of sensordata produced by sensors (93) of the vehicle (20) and/or capturedparameters of the vehicle (20).
 5. The method according to claim 1,wherein a driver assistance system (94) controls a function of thevehicle (20) on the basis of the output images.
 6. The method accordingto claim 1, wherein the output images are displayed to a driver of thevehicle (20) on a display apparatus.
 7. The method according to claim 1,wherein the region relevant for driving is marked in the storedthree-dimensional surroundings model (30) on the basis of sensor dataproduced by sensors (93) of the vehicle (20) and/or captured parametersof the vehicle (20).
 8. A vehicle control system (90) of a vehicle (20)for producing images of a stored three-dimensional model (30) of thesurroundings of the vehicle (20), said images having been corrected forperspective, having: a camera device (21) which is configured to produceat least one camera picture (32); a computing device (91) which isconfigured: to project the produced camera picture (32) onto aprojection surface (31) in the stored three-dimensional model (30) ofthe surroundings of the vehicle (20), to mark a region (33) relevant fordriving in the stored three-dimensional surroundings model (30), toproject the marked region onto a corresponding projection surface area(35) of the projection surface (31), to identify an image region of thecamera picture (32) projected onto the projection surface (31)corresponding to the marked region (33) as having been corrected forperspective, and to produce an image of the projection surface (31) withthe region (33) projected onto the projection surface area (35) by meansof a virtual camera (72) that can move freely in the surroundings model(30); and an output device (92) which is configured to output theproduced image.
 9. The vehicle control system (90) according to claim 8,wherein the computing device (91) is further configured to project themarked region (33) onto the projection surface (31), in that it images arespective point (37) of the marked region (33) onto an intersection(39) of a corresponding connecting line (38) with the projection surface(31), wherein the connecting line (38) connects this point (37) of themarked region (33) to a predefined reference point (34) of thesurroundings model (30).
 10. The vehicle control system (90) accordingto claim 9, wherein the camera device (21) has at least one vehiclecamera (21 a-21 d), and wherein the reference point (34) of thesurroundings model (30) corresponds to a spatial position of thatvehicle camera which produces the camera picture (32) imaged onto theprojection surface (31).
 11. The vehicle control system (90) accordingto claim 8, having at least one sensor (93) of the vehicle (20) forproducing sensor data and/or for capturing parameters of the vehicle(20); wherein the computing device (91) is further configured to markthe region relevant for driving in the stored three-dimensionalsurroundings model (30) on the basis of the produced sensor data and/oron the basis of the captured parameters of the vehicle (20).
 12. Thevehicle control system (90) according to claim 11, wherein the computingdevice (91) is further configured to determine a camera position and/orcamera alignment of the freely movable virtual camera (72) on the basisof the produced sensor data and/or on the basis of the capturedparameters of the vehicle (20).
 13. The vehicle control system (90)according to claim 8, having a driver assistance system (93) which isconfigured to control a function of the vehicle (20) on the basis of theoutput images.
 14. The vehicle control system (90) according to claim 8,wherein the output device (92) comprises a display apparatus, on whichthe output images can be displayed to a driver of the vehicle (20). 15.A vehicle (20) having a vehicle control system (90) according to claim8.